DNN: A Distributed NameNode Filesystem for Hadoop

The Hadoop Distributed File System (HDFS) is the distributed storage infrastructure for the Hadoop big-data analytics ecosystem. A single node, called the NameNode of HDFS stores the metadata of the entire file system and coordinates the file content placement and retrieval actions of the data storage subsystems, called DataNodes. However the single Na-meNode architecture has long been viewed as the Achilles' heel of the Hadoop Distributed file system, as it not only represents a single point of failure, but also limits the scalability of the storage tier in the system stack. Since Hadoop is now being deployed at increasing scale, this concern has become more prominent. Various solutions have been proposed to address this issue, but the current solutions are primarily focused on improving availability, ignoring or paying less attention to the important issue of scalability. In this thesis, we first present a brief study of the state-of-art solutions for the problem, assessing proposals from both industry and academia. Based on our unique observation of HDFS that most of the metadata operations in Hadoop workload tend to have direct access rather than exploiting locality, we argue that HDFS should have a flat namespace instead of the hierarchical one as used in traditional POSIX-based file system. We propose a novel distributed NameNode architecture based on the flat namespace that improves both the availability and scalability of HDFS, using the well-established hashing namespace partitioning approach that most existing solutions avoid to use because of the loss of hierarchical. We also evaluate the enhanced architecture using a Hadoop cluster, applying both a micro metadata benchmark and the standard Hadoop macro benchmark. iii Acknowledgment I would like to thank my advisor, Dr. Hong Jiang for his invaluable guidance and support over the past few years. He led me into this wonderful research area, taught me how to do research, and how important critical thinking is. I would like to thank Dr. David Swanson and Dr. Ying Lu for serving as my committee member and reviewing my thesis. I would like to thank all the members of the ADSL lab for their support and suggestions. I would like to express my special thanks to my friend Lei Xu, for his help, guidance, and friendship over the years. I thank all my collaborators of the Advanced Development & Architecture team at NetApp Inc. as this work started when I was an summer intern in the team. …